Haptic Feedback Microscope

ABSTRACT

A system and method for using a microscope to at least haptically observe a specimen in a fluid is provided. In one embodiment of the present invention, an audio frequency modulation sensing (AFMS) device is used to convert an optical signal from the specimen into an electrical signal. A haptic feedback device is then used to convert the electrical signal in at least vibrations, thereby providing a user with haptic feedback associated with the optical signal from the specimen. In another embodiment, a second electrical signal can be provided to a second haptic feedback (e.g., shaker, piezo electric, electric current inducing, etc.) device in the fluid, thereby allowing for bidirectional haptic feedback between the user and the specimen. In other embodiments, aural data can be extracted from the electrical signal and presented to the user either alone in in synchronization with video data (e.g., from a video camera).

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to a system and method for using amicroscope to at least aurally observe a specimen in a fluid (e.g.,water, air, etc.), or more particularly, to a microscope that has beenmodified to include a remote frequency modulation sensing (AFMS) devicefor capturing a visual of the specimen and extracting audio frequencymodulated electromagnetic energy therefrom.

2. Description of Related Art

The compound microscope was invented in the early 1600s, allowingscientists, for the first time, to see things that were theretofore notknown to exist. For example, the compound microscope allowed scientiststo confirm the existence of cells, study microorganisms, and examineplants, animals, and fungi to a level that before was thought to beimpossible. Even today, the microscope is one of the most importanttools used in hospitals, clinics, and laboratories across the country.

Microscopes can produce stunning images of organisms, such as protozoa,fauna, flora, metazoa, and ciliates. However, there are two majordrawbacks with such images. First, the images are generally of theorganism taken out of their natural environment. For example, becauseorganisms are generally active and move rapidly and erratically in athree dimensions space, they are usually captured, chemically stained(e.g., using iodine, ethidium bromide, etc.), and mounted onto amicroscope slide before they are viewed. By way of another example,methylcellulose and/or other thickening chemicals may be added, and/orthe organism may be squeezed between glass plates to slow the organismdown for better viewing and reduced depth of field requirements. Whilesuch techniques allow for easier viewing, they do not allow for in situviews of healthy microscopic organisms that are naturally swimmingwithin their aquatic micro-environments.

Second, while compound microscopes allow organisms to be visuallyobserved, they do not provide for any aural observation. In other words,even though biologically active creatures produce sound energy, a resultof both their internal and external motions, there is no currentsolution for observing these sounds. While millions (if not billions) ofpeople have observed microscopic inhabitants in detail since themicroscope was invented, fewer people have observed the inhabitantswhile they are still active in their surroundings, and very few people(if any) have listened to the sound energy produced by these inhabitantsin their microscopic environments.

In our normal world sound plays an important and complementary role tosight. Since the introduction of the sound movie in 1927, andstereophonic sound movies in the late 1940's, no major feature film hasbeen released without multichannel sound as an essential and fundamentalelement. The cinematic ability to see and hear detailed and dynamicallychanging microscopic scenery would lead to a better and more immersiveunderstanding of these amazing very alien micro-worlds. Additionalfeatures (e.g., haptic feature, tracking feature, optical stainingfeature, etc.) potentially provide a most interesting and unusual way ofobserving and/or interacting with the micro-aquatic organisms andenvironments.

Thus, in light of the foregoing, it would be advantageous to develop asystem and method that overcame as least some of the foregoingdrawbacks, and allowed for at least aural observation of a microscopicorganism in its natural environment.

SUMMARY OF THE INVENTION

The present invention provides a system and method for using amicroscope to at least aurally observe an organism in a fluid (e.g.,water, air, etc.). Preferred embodiments of the present inventioninclude a standard compound microscope modified to capture visuals ofthe specimen, extract aural data from the visuals, and use the auraldata to provide an audio output. Visuals may also be used to provide avideo/image output in time synchronization with the audio output.

In one embodiment of the present invention, the microscope is modifiedto include a first beam splitter, splitting the visual of the specimenmagnified by the objective lens (i.e., the optical signal) into twooptical signals. Once the beam is split, a first beam travels upward. Asecond beam, however, is provided to an audio frequency modulationsensing (AFMS) device, whose function is to sense photoacousticmodulation of the specimen, which is accomplished through at least onesensor. In other words, at least one sensor is used to convert anoptical signal from the specimen into an electrical signal. The audiofrequency modulated electromagnetic energy in the electrical signal isthen extracted (e.g., amplified, etc.), allowing the sound energy to beobserved by a user (e.g., displayed on a screen, played on a speaker,etc.).

In this embodiment, the first beam (from the first beam splitter) maytravel upward into a second beam splitter, where the first beam is splitinto two separate beams (or optical signals). The first beam is providedto the eyepiece, allowing the user to visualize the specimen in realtime. The second beam is provided to at least one other sensor, where asecond visual of the specimen is captured. The visual can then bedisplayed on a screen in time synchronization with the aural data.

In another embodiment of the present invention, only one beam splitteris required. This is because a single sensor (or single set of sensors),e.g., as included in a digital video camera, is used to capture visualsof the specimen. The captured visuals are then provided to the display,e.g., allowing the visuals to be displayed to the user, and provided tothe AFMS device, e.g., allowing aural data to be extracted. The auraldata is then output and used as discussed above (e.g., in timesynchronization with the video data, etc.). Such an embodiment isadvantageous in that it only requires a single sensor (or set ofsensors) to capture visuals of the specimen. The captured visuals canthen be used to both generate video and audio outputs. And if a digitalvideo camera is used, and different pixels (or sets of pixels) are usedto capture different visuals, stereophonic (or multi-channel) sound canbe generated, providing a cinematic, multimedia experience for the user.

In the foregoing embodiments, the beam splitter in the single beamsplitter embodiment, and the second beam splitter in the dual beamsplitter embodiment, are used to provide visuals of the specimen in realtime to the user via the eyepiece. It should be appreciated, however,that the eyepiece, and therefore the beam splitter in the single beamsplitter embodiment, and the second beam splitter in the dual beamsplitter embodiment, are not limitations of the present invention. Forexample, a microscope where a digital camera is used to capture visualsof the organism, where the visuals are then provided (e.g., via a videooutput) to a display (e.g., an LCD display, etc.), allowing the user tovisualize the organism in real time (i.e., by watching the display), iswithin the spirit and scope of the present invention.

As discussed above, one objective of the present invention is to vieworganisms in their natural, aquatic environment. However, because soundenergy produced by these organisms is extremely low, and thereforedifficult to capture and extract, the inventor has discovered that useof a water immersive objective lens is advantageous to the presentinvention. A water immersive objective is a specially designed objectivelens used to increase the resolution of the microscope. This is achievedby immersing both the lens and the specimen in water, which has a higherrefractive index (˜1.33) than air (˜1.0003), and a similar refractiveindex to most living cells (˜1.35), thereby increasing the numericalaperture of the objective lens. By moving the objective below the watersurface, we can also eliminate visual distortions resulting from waterripple.

The inventor has also discovered that optical staining, such asdark-field, is advantageous to the present invention. Dark-fieldtechniques can be achieved, for example, using a block that prevents theillumination from directly entering the objective lens, only allowingreflected or scattered light to enter the objective lens. This has theresult of obliquely illuminating the specimen allowing the acousticenergy to be sensed with an improved signal to noise ratio. Similarbenefits can be achieved using other optical staining techniques, asdiscussed in greater detail below.

Because certain organisms move rapidly, a tracking system may be used toposition (or maintain) the organism under the objective lens. Forexample, a manual tracking system may be employed that includes a clampthat goes around a container (e.g., Petri dish) that houses the organism(not shown), where the container is supported vertically via a stage.While knobs may be used to move the stage, and therefore the container,in the Z direction, the clamp may be used to move the container in atwo-dimensional space (e.g., in the X and/or Y directions). The trackingsystem may include a handle configured to be gripped by the user andused to move the container in relation to the stage and/or objectivelens.

In other embodiment, the tracking system may be motorized and/orautomated. For example, in one embodiment, a joystick may be used tocontrol a plurality of motors, which are used to move the container inrelation to the objective lens. In another embodiment, softwareoperating on the CPU may be used to automatically track a specimenwithin the container. This may be accomplished, for example, by usingsoftware to monitor images of a specimen. If the software detectsmovement, motors can be controlled to reposition the specimen withrespect to the objective lens. The foregoing can be used to move thecontainer within a two-dimensional space (e.g., in the X and/or Ydirections) or within a three-dimensional space (e.g., in the X, Y,and/or Z directions).

A more complete understanding of a system and method for using amicroscope to at least aurally observe a specimen in a fluid (e.g.,water, air, etc.) will be afforded to those skilled in the art, as wellas a realization of additional advantages and objects thereof, by aconsideration of the following detailed description of the preferredembodiment. Reference will be made to the appended sheets of drawings,which will first be described briefly.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a prior art compound microscope, which can be used tovisually observe a specimen;

FIG. 2A illustrates a compound microscope modified in accordance withone embodiment of the present invention, which can be used to bothvisually and aurally observe a specimen;

FIG. 2B illustrates a compound microscope modified in accordance withanother embodiment of the present invention, which can be used to bothvisually and aurally observe a specimen;

FIG. 2C illustrates a compound microscope modified in accordance withyet another embodiment of the present invention, which can be used toboth visually and aurally observe a specimen;

FIGS. 2D-F illustrate alternate embodiments of the present invention,where light is projected upward, toward an AFMS device that is above thespecimen, downward toward an AFMS device that is below the specimen, andfrom the side, allowing reflective light to be captured by an AFMSdevice that is either above or below the specimen;

FIG. 2G illustrates an embodiment of the present invention where lightfrom a point source illuminator and/or dark field illumination isreceived (either directly or indirectly) (e.g., reflected from orscattered by the test cell) by a photodetector array and used togenerate a single or multichannel audio signal;

FIG. 3 illustrates a beam splitter that can be used to generate atransmit beam (e.g., first optical signal) and a reflected beam (e.g.,second optical signal) from an input beam (e.g., received opticalsignal);

FIG. 4 illustrates a water immersion objective, allowing both theobjective lens and the specimen to be immersed in a fluid, such aswater;

FIG. 5 illustrates a microscope using both bright-field and dark-fieldtechniques;

FIG. 6 illustrates how dark-field techniques block light from directlyentering the objective lens;

FIG. 7 compares specimens that are being viewed using both bright anddark-field techniques;

FIG. 8 illustrates organisms visually observed using an embodiment ofthe present invention;

FIG. 9 illustrates organism visually and aurally observed using anembodiment of the present invention;

FIG. 10 illustrates a block diagram of one embodiment of the presentinvention;

FIGS. 11 and 12 illustrate tracking features included in certainembodiments of the present invention;

FIG. 13 illustrates an AFMS device in accordance with one embodiment ofthe present invention;

FIG. 14 provides a method for modifying a microscope to allow for atleast aural observations in accordance with one embodiment of thepresent invention; and

FIG. 15 provides a method for operating the microscope described in FIG.14 in accordance with one embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Preferred embodiments of the present invention include a microscopemodified to capture first and second visuals of a specimen, where auraldata is extracted from the second visual of the specimen. Video datafrom the first visual is then output in time synchronization with theaural data. It should be appreciated that while the present invention isdescribed in terms of a compound microscope being modified to at leastaurally observe an organism, the present invention is not so limited.Features described herein can be used in conjunction with any viewingdevice (e.g., telescope, binoculars, magnifying glass, etc.), with orwithout optics (e.g., objective lens, etc.), with or without opticalstaining techniques (e.g., dark-field, shadowgraph, etc.), to aurallyobserve any specimen (e.g., cells, insects, animals, plants, fungi,etc.).

A prior art compound microscope 100 is shown in FIG. 1. Such amicroscope can generally be broken down into three portions, i.e., thehead/body; the base; and the arm. The head/body houses the optical partsin the upper part of the microscope, including an eyepiece (or ocularlens) 102, a diopter adjustment 104, a nose piece 128, and an objectivelens 126. The base 114 of the microscope 100 houses a light source (ormirror) 116, and may include an on/off switch 112. The arm 106 connectsthe base 114 to the head/body, and generally includes a stage 122, slideholders 124, a condenser 120, and an iris diaphragm 118.

The eyepiece (or ocular lens) 102 is what a user looks through at thetop of the microscope 100. Typically, standard eyepieces have amagnifying power of 10×. Optional eyepieces of varying power areavailable, typically from 5×-30×. If there are two eyepieces, thediopter adjustment 104 can be used to adjust for inconsistencies betweenthe user's left and right eye. The nosepiece 128 houses the objectivelens 126, which is the primary optical lens on the microscope 100.Objective lenses 126 generally range from 4×-100×, and can be eitherforward or rear-facing. The stage 122 is where the specimen is placed,and includes an aperture, allowing light from the light source 116 toreach (e.g., illuminate) the specimen. The condenser 120 is used tocollect and focus the light from the light source 116 onto the specimen,and the iris diaphragm 118 is used for light control and changing theillumination angle. Light intensity is generally controlled by the irisdiaphragm 118 when it is well removed from the conjugate focus location,and illumination angle is controlled when the illumination position isat the focus of the iris diaphragm 118. In certain microscopes, two irisdiaphragms (not shown) are used, where the first one controls lightintensity and the second one controls illumination angle. The on/offswitch 112 provides power to the light source 116 (and/or otherelectrical features on the microscope 100), and the knobs 108, 110 areused to focus the microscope on the specimen.

A first embodiment of the present invention is shown in FIG. 2A. Whilethe microscope depicted in FIG. 2A is like the one shown in FIG. 1 inthat it includes a light source, a stage, an objective lens, and aneyepiece 208, it further includes features that allow the specimen to beboth visually and aurally observed. For example, the microscope mayinclude a first beam splitter 202, splitting the visual of the specimenmagnified by the objective lens (i.e., the optical signal) into twooptical signals. Such a beam splitter can be seen in FIG. 3. Ingenerally, it functions by taking an input beam, passing (ortransmitting) a portion thereof, and reflecting the remaining portion.It can be constructed using a cube made from two triangular glassprisms, which are glued together at their base. The thickness of theresin layer is adjusted such that half of the light incident through one“port” is reflected, and the other half is transmitted due to frustratedtotal internal reflection. It should be appreciated that the presentinvention is not limited to any particular beam splitter, and all typesand constructions generally known to those skilled in the art (e.g., apolarizing beam splitter, those involving pellicle mirrors, thoseinvolving dichroic optical coatings, etc.) are within the spirit andscope of the present invention.

Once the beam is split, a first beam travels upward, and will bediscussed in detail below. The second beam, however, is provided to anaudio frequency modulation sensing (AFMS) device 204, whose function isto capture audio frequency modulated data from the specimen, which isaccomplished through at least one sensor (e.g., one or morephotodetector, pixel, solar array, solar cell, etc.). In other words, atleast one sensor is used to convert an optical signal from the specimeninto an electrical signal. The audio frequency modulated electromagneticenergy in the electrical signal is then extracted, allowing the soundenergy to be observed by a user (e.g., displayed on screen, convertedinto mechanical energy using at least one transducer, etc.). It shouldbe appreciated that the term “photodiode” is used herein in its broadsense, to include any device that converts an optical signal (or portionthereof) into an electrical signal. It should also be appreciated thatthe term “extracted” (and variations thereof) is used herein in itsbroad sense, to include any device that allows, aids, or enhances theobservation of sound energy included in the signal provided by thephotodiode.

By way of example only, FIG. 13 shows an exemplary AFMS device, whichincludes at least one photodiode 1300 for converting an optical input(e.g., magnified visual of the specimen, etc.) into electrical energy,and an extraction circuit for providing an audio output (e.g., an outputcomprising aural data). The extraction circuit may include apreamplifier 1310 for amplifying the electrical signal, and may furtherinclude at least one filter (e.g., low-pass, high-pass, band-pass,band-stop, notch, adaptive, etc.) 1322 for removing unwantedfrequencies, artifacts, noise, etc., and at least one other amplifier1324. It should be appreciated that the post-processing circuitry 1320may either be included in the AFMS device, or included in a separatedevice, constructed especially for enhancing sound (e.g., a studio soundrecorder, a computer running sound recording software, etc.) (notshown). It should also be appreciated that the present invention is notlimited to the AFMS device shown in FIG. 13. Other AFMS devices thatinclude additional, different, or fewer components are within the spiritand scope of the present invention. For example, different AFMS devicesare discussed and depicted in a patent application entitled “Solar ArrayRemote Acoustic Sensing (SARAS),” filed on Aug. 6, 2016, bearing Ser.No. 15/230,369. Those devices, including their construction and use(see, e.g., FIG. 20), are specifically incorporated herein by reference.

Referring back to FIG. 2A, once aural data is extracted from themagnified visual of the specimen, the aural data can be provided to arecording device 214 and/or at least one transducer, such as those foundin speakers 216. By providing aural data to the speakers 216, the auraldata can be converted into sound, allowing the user to aurally observe(e.g., listen to) the specimen. By providing aural data to the recordingdevice 214, aural data can be recorded and played back at a subsequenttime (e.g., in time synchronization with video data). As previouslydiscussed, the recording device can also be used to enhance the auraldata (e.g., filter, amplify, etc.) before it is played for the user. Itshould be appreciated that the present invention is not limited to auraldata being distributed as shown in FIG. 2A. For example, aural data mayonly be provided to a speaker, may only be provided to a sound recorder,or may be provided to a speaker via the sound recorder. By way of yetanother example, aural data may only be provided to a computer, allowingit to be stored, enhanced, or displayed on a screen (see, e.g., FIG. 9).It should also be appreciated that aural data may be extracted fromvisuals captured by a single sensor, resulting in monophonic (or singlechannel) sound, or it may be extracted from visuals captured by aplurality of sensors (e.g., a plurality of photodiodes, a plurality ofpixels, etc.), resulting in stereophonic (or multi-channel) sound. Thisfeature is discussed at length in Ser. No. 15/230,369 (see, e.g., p. 25,I. 1-p. 26, I. 6), and is specifically incorporated herein by reference.

As previously discussed, the first beam splitter 202 results in twobeams, one of which travels upward into a second beam splitter 206. Thesecond beam splitter is like the first beam splitter in that it splitsan input beam (e.g., a visual of the specimen magnified by the objectivelens) into two optical signals. The first optical signal is provided tothe eyepiece, allowing the user to visualize the specimen in real time.The second optical signal is provided to at least one other sensor 210,where a second visual of the specimen is captured. This visual can thenbe displayed on a screen 212 (e.g., a computer screen, an LCD monitor, aplasma television, etc.) and/or provided to a digital video recorder(DVR) 214.

It should be appreciated that a single recording device can be used torecord and/or modify both video and audio data (as shown), or a firstrecording device can be used to record and/or modify video and a secondrecording device (not shown) can be used to record and/or modify audio.It should also be appreciated that the present invention is not limitedto video data being distributed as shown in FIG. 2A. For example, videodata may only be provided to a display, may only be provided to a DVR,or may be provided to a display via the DVR. By way of yet anotherexample, video data may only be provided to a computer, allowing it tobe stored, adjusted, or displayed on a screen (see, e.g., FIG. 9). Itshould further be appreciated that the present invention is not limitedto the use of at least one other sensor 210, as shown in FIG. 2A. Forexample, a single sensor (or set of sensors) may be used to capturevisuals for both audio and video data, as shown in FIG. 2B (discussedbelow). By way of yet another example, the eyepiece 208 may be the onlymeans for the user to view the specimen during aural observation.

It should also be appreciated that, if at least one other sensor is usedto capture a second visual of the specimen, the sensor can be any sensorgenerally known to those skilled in the art, including, but not limitedto those found in digital video cameras. The visuals captured can beimages or video, can be captured using any frame rate (e.g., 240 FPS issufficient to capture audio generated by most microorganisms (˜50 Hz),at least 4000 FPS (e.g., a high-speed digital camera) may be necessaryto provide an adequate audio bandwidth while being sensitive to weakoptical modulation levels, etc.). Different devices for capturingvisuals are discussed and depicted in Ser. No. 15/230,369 (see, e.g., p.18, I. 29-p. 19, I. 20, and p. 25, I. 26-p. 26, I. 6), and arespecifically incorporated herein by reference.

A second embodiment of the present invention is depicted in FIG. 2B.This embodiment is substantially the same as the embodiment depicted inFIG. 2A except that only a single beam splitter 206 is needed. This isbecause a single sensor (or single set of sensors) 210 (e.g., asincluded in a digital video camera) is used to capture visuals of thespecimen, as magnified by the objective lens. The captured visuals arethen provided to the display 212 (e.g., allowing the visuals to bedisplayed to the user), to the DVR 214 (e.g., allowing the visuals to berecorded and/or adjusted), and to the AFMS device 204 (e.g., allowingaural data to be extracted). The aural data is then output and used asdiscussed above (e.g., in time synchronization with the video data,etc.) (e.g., where it is recorded, adjusted, played, etc.). Such anembodiment is advantageous in that it only requires a single sensor (orset of sensors) to capture visuals of the specimen. The captured visualscan then be used to generate both video and audio outputs. And if adigital video camera is used, and different pixels (or sets of pixels)are used to capture different visuals, stereophonic (or multi-channel)sound can be generated, providing a cinematic, multimedia experience forthe user. A third embodiment of the present invention is depicted inFIG. 2C. This embodiment is substantially the same as the embodimentdepicted in FIG. 2B except that it does not require a beam splitter.Instead, a single sensor (or single set of sensors) 210 (e.g., asincluded in a digital video camera) is used to capture visuals of thespecimen, as magnified by the objective lens. The captured visuals arethen provided to the display 212 (e.g., allowing a user to visualize theorganism in real time), to the DVR 214 (e.g., allowing the visuals to berecorded and/or adjusted), and to the AFMS device 204 (e.g., allowingaural data to be extracted). The aural data is then output and used asdiscussed above (e.g., in time synchronization with the video data,etc.) (e.g., where it is recorded, adjusted, played, etc.). Such anembodiment is advantageous in that it only requires a single sensor (orset of sensors) to capture visuals of the specimen, and does not requirea beam splitter. The captured visuals can then be used to generate bothvideo and audio outputs. And if a digital video camera is used, anddifferent pixels (or sets of pixels) are used to capture differentvisuals, stereophonic (or multi-channel) sound can be generated,providing a cinematic, multimedia experience for the user.

It should be appreciated that while the present invention has beendescribed in terms of illuminating a specimen from below, and using anAFMS device (e.g., together with an objective lens) to capture visualsof the specimen, the present invention is not so limited. For example,as shown in FIG. 2D, a specimen 222 (e.g., an organism, etc.) in itsnatural environment 220 (e.g., water, etc.) can be illuminated by alight source (e.g., an LED, etc.) from below, and an AFMS device 204located above the specimen 222 can be used to receive optical signalsand to extract aural data therefrom. By way of another example, as shownin FIG. 2E, the specimen 222 (e.g., an insect, etc.) in its environment220 (e.g., air, etc.) can be illuminated by a light source 218 (e.g.,the sun, etc.) from above, and an AFMS device 204 located below thespecimen 222 can be used to receive optical signals and to extract auraldata therefrom. By way of yet another example, as shown in FIG. 2F, thelight source 218 (point source or otherwise) can be positioned to theside of the specimen 222, allowing only (or primarily) reflected (orrefracted) light to be captured by the AFMS device 204.

Such an embodiment has advantages that are discussed in greater detailbelow, with respect to optical staining techniques. In eitherembodiment, the visuals can be magnified, or not (e.g., using anobjective lens, etc.), depending on the specimen that is being observed.

In yet another embodiment, the system may include combinations of theforegoing. For example, as shown in FIG. 2G, a microscopic test cell 222(or some other organism, insect, etc.) may be illuminated in a way thatmaximizes the acousto-optic modulation depth. This may be accomplished,for example, using a point of source illuminator 228 (e.g., using“shadowgraph” techniques, etc.) and/or a dark-field illuminator 238(e.g., dark-field oblique illumination, laser scatter, etc.). Aphotodetector 224 (e.g., a photodiode, such as a solar cell, formonophonic, a photodiode array, such as a solar array, for stereophonic,etc.) could then be used to convert the received optical signal into anelectrical signal, and at least one audio amplifier 234 could be used toamplify at least an audio frequency component included therein. Asdiscussed above, other circuitry could be used to “clean up” the auraldata. A video output could also be generated (e.g., using the samearray, a different array, etc.). Again, an objective lens (not shown)may or may not be used, depending on the specimen that is beingobserved.

As discussed above, one objective of the present invention is to vieworganisms in their natural, aquatic environment. However, because soundenergy produced by organisms is extremely low, and therefore difficultto capture and extract, the inventor has discovered that use of a waterimmersive objective lens is advantageous to the present invention. Awater immersive objective is a specially designed objective lens used toincrease the resolution of the microscope. This is achieved by immersingboth the lens and the specimen in water, which has a higher refractiveindex (˜1.33) than air (˜1.0003), and a similar refractive index to mostliving cells (˜1.35), thereby increasing the numerical aperture of theobjective lens. Also, by moving the objective below the water surface,visual distortions resulting from water ripple (e.g., defocusing,deforming, etc.) can be eliminated.

Such a lens is shown in FIG. 4, where the lens 404 is immersed in thesame fluid 402 as the specimen 406. In this figure, a dish 400 (e.g.,Petri dish) is used to house both the fluid 402 and the specimen 406.Water immersive lenses are manufactured by companies such as Olympus,Zeiss, Leica, and Nikon, and may include a nose (or dipping cone)housing the front lens, which is tapered at (roughly) a 43-degree angleto allow a steep inclination of approach, providing easy access to thespecimen. These objectives (typically) also have a long working distanceranging from 2.0 mm (60× Pan Fluorite) to 3.3 mm (10×-40× PlanFluorite), which are useful in allowing additional room formicromanipulation of the specimen. It should be appreciated that whilecertain water immersive objectives have been discussed herein, thepresent invention is not limited to any particular objective, waterproofor otherwise.

And because the light reflecting off (or illuminating) the specimen isthe optical signal of interest (e.g., includes the audio frequencymodulated electromagnetic energy), the inventor has also discovered thatoptical staining, such as dark-field, is advantageous to the presentinvention. Dark-field techniques can be achieved, for example, using a“patch stop” (see FIG. 6), which block direct light 602 from enteringthe objective lens. See, e.g., FIG. 6 (showing that the direct light 604that is passed through the condenser lens does not enter the objectivelens). In dark-field, the light received by the objective lens is notdirect light 602, 604, but instead reflected light 606 from thespecimen. This is further illustrated in FIG. 5, where dark-field (e.g.,using a “patch stop” 500) prevents direct light 502 from entering theobjective lens 504, whereas bright-field allows direct light 502 toenter the objective lens 504. Slides using bright and dark-fieldtechniques can be seen in FIG. 7, with slide 702 using bright-fieldtechniques, and slide 704 using dark-field techniques. As can be seen,in dark-field, it is the specimen(s) that is illuminated, allowing soundenergy of interest to be more easily acquired.

It should be appreciated that while dark-field techniques have provenadvantageous, the present invention is not so limited. For example,other optical techniques generally known to those skilled in the art(e.g., phase contrast dispersion staining, Becke' line dispersionstaining, oblique illumination dispersion staining, objective stopdispersion staining, Schlieren techniques, shadowgraph techniques,etc.), or combinations thereof, are within the spirit and scope of thepresent invention (a greater discussion of this is provided below). Itshould also be appreciated that use of the present invention is notlimited to the production of multimedia content, or creating amicroorganism cinematic experience. By way of example only, becausedifferent specimens produce different sounds, acquisition and analysisof sound energy could prove useful in the field of cytometry, or tootherwise classify organisms found in particular environments.

Because certain organisms move rapidly, a tracking system may be used toposition (or maintain) the organism under the objective lens. Forexample, as shown in FIG. 11, a manual tracking system may be employedthat includes a clamp 1120 that goes around a container 1110 (e.g.,Petri dish) that houses the organism (not shown), where the container issupported vertically via a stage 1100. While knobs (e.g., course andfine focus knobs, as previously discussed) may be used to move the stage1100, and therefore the container 1110, in the Z direction, the clamp1120 may be used to move the container 1110 in a two-dimensional space(e.g., in the X and/or Y directions). The tracking system may include ahandle 1122 affixed perpendicularly to the clamp 1120, where the handle1122 is configured to be gripped by the user and used to move thecontainer 1110 in relation to the stage 1100 and/or objective lens (notshown).

In other embodiment, the tracking system may be motorized and/orautomated. In one embodiment, as shown in FIG. 12, a joystick 1210 maybe used to control a plurality of motors 1202, 1204, which are used tomove the container 1110 in relation to the objective lens (not shown).This may be accomplished by using at least a plurality of motors 1202,1204, an upper stage 1104, and a lower stage 1104, where the upper stage1104 is configured to move horizontally in relation to the lower stage1102. For example, by driving the first motor 1202 in a first direction,a first gear (e.g., first worm gear) (not shown), can be used to movethe upper stage 1104 in the positive X direction, and by driving thefirst motor 1202 in a second direction, the first gear can be used tomove the upper stage 1104 in a negative X direction. Similarly, bydriving the second motor 1204 in a first direction, a second gear (e.g.,second worm gear) (not shown), can be used to move the upper stage 1104in a positive Y direction, and by driving the second motor 1204 in asecond direction, the second gear can be used to move the upper stage1104 in a negative Y direction. In this embodiment, the electricalsignals from the joystick 1210 can be provided directly to the motors1202, 1204, or can be provided to a CPU 1220. With respect to thelatter, the CPU 1220 may be configured to generate signals forcontrolling the motors 1202, 1204 based on signals received from thejoystick 1210.

In another embodiment, as shown in FIG. 12, software operating on theCPU 1220 may be used to automatically track an organism within thecontainer 1110. This may be accomplished, for example, by using softwareto monitor images of the organism (see, e.g., FIG. 2A, Ref. No. 212). Ifthe software detects movement in the positive X direction, then the CPU1220 may drive the first motor 1202 in the first direction, and if thesoftware detects movement in the negative X direction, then the CPU 1220may drive the first motor 1202 in the second direction. Similarly, ifthe software detects movement in the positive Y direction, then the CPU1220 may drive the second motor 1204 in the first direction, and if thesoftware detects movement in the negative Y direction, then the CPU 1220may drive the second motor 1204 in the second direction. Such softwaremay also be used to digital tracking an organism if a digital zoom isbeing used to focus (or zoom in) on the organism within a largerenvironment. It should be appreciated that the present invention is notlimited to any particular tracking system, or the use of any trackingsystem at all. It should also be appreciated that any tracking systemused can be configured to track at least one specimen in atwo-dimensional space (e.g., X and/or Y) or in a three-dimensional space(e.g., X, Y, and/or Z), depending on design requirements.

A method for modifying a microscope in accordance with one embodiment ofthe present invention is shown in FIG. 14. Starting at step 1400, themicroscope is modified by adding a visual capture feature at step 1402.As discussed above, this may be a sensor that is dedicated for auralobservation, or a sensor that allows for both aural and visualobservations. In step 1404, a sound extraction feature is added. Asdiscussed above, this may include a pre-amplifier, a filter, etc. Thefunction of the sound extraction feature is to receive an optical signal(or an electrical conversion thereof) and to provide (e.g., output) atleast aural data. As discussed above, the aural data can then bedisplayed, played, etc. Optical staining, such as dark-field staining,may be added at step 1406. A water immersive objective lens may then beadded at step 1408. A dish (e.g., Petri dish) containing at least onespecimen in a fluid is then placed on the stage at step 1410, ending themethod at step 1412. It should be appreciated that the present inventionis not limited to method shown in FIG. 14, and that fewer, different,and/or additional steps are within the spirit and scope of the presentinvention. For example, a method that does not involve optical staining,or involves multiple visual capture features, is within the spirit andscope of the present invention.

Once the microscope is constructed, it can operate as shown in FIG. 15.For example, starting at step 1412, the water immersive objective lensmay be lowered into the fluid at step 1500. Once the lens isfocused/aligned (e.g., using the focus and/or tracking techniquesdiscussed above), at least one visual of the specimen is captured atstep 1502. As discussed above, this may be a visual that is dedicated toaural observation or a visual that is used for both aural and visualobservations. Aural data is then extracted from the visual data at step1504 and output in time synchronization with video (or image) data atstep 1506, ending the method at step 1508. It should be appreciated thatthe present invention is not limited to the method shown in FIG. 15, andthat fewer, different, and/or additional steps are within the spirit andscope of the present invention.

EXPERIMENTAL OBSERVATIONS OF CERTAIN EMBODIMENTS

In testing the present invention, the inventor constructed a device thatfollowed (basically) the block diagram illustrated in FIG. 10, whichproduced the results shown in FIGS. 8 and 9. FIG. 8 shows a plurality oforganism in their natural environment, certain ones of which havingcilia 800. FIG. 9 shows the same cilia in two second intervals over aneight second period, going from retracted to spinning. FIG. 9 also showssound energy produced by the cilia spinning during a two and a halfsecond period. The following are the testing procedures that were usedand the inventor's experimental observations.

Free swimming protozoa (by way of example) and other bio-activityproduces localized sounds, motions and vibrations that weakly modulatethe ambient illumination within a micro-aquatic environment. An opticalsensor can detect these faint audio frequency optical intensitymodulations and can be made sensitive to phase, polarization, scatteringand other optical phenomena through the use of microscopic opticalstaining techniques. Optical staining significantly improves both thevisual image quality and aural sensitivity. With localized acousto-opticsensitivity, monophonic or stereophonic (multichannel) sound can beextracted and synchronized with high definition video to create a moreimmersive cinema view into these aquatic micro-environments. Much of themicro-acoustic activity is relatively low frequency, spatially resolvedand well suited to multichannel haptic and tactile feedback. Examples ofvisual and aural micro-aquatic observations are provided herein.

The imaging aspect of this microscope is conventional although optimizedfor this application. Imaging enhancements include (but do not require)a high definition video camera, optical staining for both visual andaural use, the use of water contact optics to better roam throughout theaquatic micro-environments and stabilize the optical staining process, afast mechanical joystick controlled XY stage for effectivemicro-organism tracking, and LED illumination to minimize thermaldistress. While LED illumination is preferred, a direct current (DC)incandescence light bulb can also be used. A DC light source ispreferred because it minimizes fluctuation, or light flicker, andprovides a superior optical signal for carrying sound energy. In certainembodiments, sunlight can also be used as the light source. See, e.g.,FIGS. 2E and 2G.

Sounds within the aquatic micro-environment are sensed using RemoteAcoustic Sensing (RAS) technology, or an AFMS device. It is believedthat this is the first use of this technology for listening to thesounds produced by aquatic micro-organisms within their environment.Some of the sounds are associated with cilia motion and propulsiveactivity. Other sounds result from various periodic oscillations andwith protozoa colliding into each other and other micro flora and fauna.Many rotifers with their spinning cilia produce low frequency dronesounds occasionally interrupted by nearby disturbances. Other sounds arecaused by very rapid body motions. As many of the micro fauna aretransparent, some sounds have been associated with protozoa within thedigestive tract of other larger metazoa. Some of the sounds are similarto aircraft, some like cars, others like colored and 1/f noise, othersmake periodic or random pulsing sounds, and others are hard to describe.The micro-organism sounds are often species specific and sometimes evenspecific to individual animals.

There are a variety of acousto-optic modulation and emission mechanisms,some of which have been previously described in the context of observingdistant aerospace vehicles. See, e.g., Ser. No. 15/230,369. In somecases, the sounds result from acoustic propagation of pressure waves andin other cases, are the result of non-acoustic optical intensity orphase variations at audio frequency rates. Several AOM mechanismsapplicable to the micro-aquatic environment include:

-   -   Absorptive AOM: Rapidly moving cilia and larger opaque or        partially opaque micro-aquatic fauna cause absorptive optical        intensity modulation at audio frequency rates. In this case, the        optical modulation is usually the direct result of the fauna        motion and not as an acoustic pressure wave although the fauna        motion does also produces a weaker propagating acoustic pressure        wave throughout the micro-aquatic environment.    -   Reflective AOM: In the presence of dark field illumination, the        micro-fauna appears as bright animals in front of a dark        background. This is due to the oblique illumination reflected by        the fauna into the microscope objective. Audio frequency energy        sources within the micro-aquatic environment modulate the        reflectivity at audio rates which then is detected as audio        frequency optical intensity variations. Air bubbles as        reflective spherical membranes provide another type of a        reflective acousto-optic modulator.    -   Refractive AOM: Protozoa and other micro-fauna are largely        transparent and mostly invisible with conventional bright field        (transmitted) illumination. Their refractive index is different        than water so both dark-field and interference staining        techniques can greatly enhance their visibility and acoustic        detectability. As the micro-fauna bodies are compressible and        compliant, their optical modulation is likely the result of        acoustic pressure waves propagating through both the water and        fauna.    -   Optical emitters: Bio luminescence may result in audio frequency        emissions. So far, audio frequency bio-luminescent emissions        have not been experimentally observed due to a lack of        specimens.

The reflective, refractive and absorptive AOMs produce audio frequencyintensity and phase modulations. RAS sensitivity is a function of thesignal level and noise background. In general, the level of the audiofrequency modulated signal component should be maximized and all otherterms (steady state illumination, sensor noise, etc.) should beminimized.

Shot noise is the result of individual photons arriving at the detector.The noise level power is related to the square root of the intensity.Shot noise is minimized by reducing steady state background illuminationthrough the use of optical staining techniques. One of the moreeffective techniques is to use dark-field illumination where only activefauna are visible. Color difference interference polarization stainingresults in more background light than dark-field but sometimes showdetails that are not visible with dark-field illumination. Placing anarrow band filter in the RAS channel when using interferencepolarization produces an exceptionally dark background, but at theexpense of some light loss due to the narrow filter passband.

The RAS detector noise is a function of the detector area, photodiodeprocess specific noise terms, Johnson thermal noise and the detectorpreamplifier design. Detector noise can be minimized by proper detectorselection and circuit design. Increasing the overall light intensitygenerally helps but this can cause fauna health and detector saturationproblems. LED illumination (while not necessary) works well as themicroscope light source, but it may be necessary to confirm that the LEDelectronics do not cause any light intensity modulation in the audiospectrum.

Other potential noise sources include optical interference, electricalinterference and mechanical vibrations of the microscope. Opticalinterference is common from fluorescent and other room lamps but can beeasily eliminated. Even unmodulated ambient daylight illumination canincrease the shot noise background. Electrical interference from thevideo camera or other electronics into the highly sensitive RASdetectors can be more difficult to control. Mechanical vibrations of themicroscope will couple into the micro-aquatic environment and be heardthrough the RAS detectors. RAS noise tests should be made with themicroscope lamp, video system, room lights, etc., each separately cycledon and off.

Maximum acoustic sensitivity is achieved by maximizing the acousticallymodulated signal component level, minimizing unmodulated detectorillumination and matching the AOM area to the detector sensitive area,ideally as a spatially matched filter. The transfer function from anacoustic source in the micro-aquatic environment to the RAS detectoroutput can be determined by a tristatic communication link model. Thethree links are the optical illumination of the AOM, the acoustic energyarriving at the AOM and the propagation path from the AOM to the opticalreceiver. The acoustic signal level is a function of the illuminationlevel, the acoustic level, the AOM modulation efficiency, microscopeoptics and the RAS detector.

Many microscopic fauna and flora are transparent and nearly invisiblewhen viewed with conventional bright-field (transmitted light)microscopes. Improving the visibility is important both for the pictureand sound elements. Chemical staining of microscopic specimens with dyesgreatly improves visibility but is not compatible with live animalviewing. A variety of optical staining techniques have been developedover many years to provide non-chemical staining alternatives. Theseprovide a wide range of visibility enhancement mechanisms that canenhance color and contrast. There are many variations but opticalstaining generally relies on optical scattering, polarization or opticalinterference phenomena.

Optical scattering or dark-field techniques involve illuminating thescene in a way that direct illumination does not enter the microscopeobjective lens. There are many variations based on the scatteringgeometry including forward, back and side scattering. Forward scatterillumination is normally called dark-field. Illumination from obliqueangles behind the object is forward scattered by the object toward theobserver who sees the bright object against a black background.Dark-field illumination used with lower power microscope objectivelenses is simple to implement, low cost and effective. A conventionalbright field Koehler illuminated condenser lens can be converted todark-field illumination simply by placing an annular mask into thecondenser lens filter holder to block the direct illumination. Somebright-field condenser lenses include an iris and lateral shiftcapability to produce an oblique forward scattered dark-fieldillumination cone at a specific offset angle. Annular, oblique andvarious intermediate forward scatter variations were used in thisproject. Other optical scattering illumination methods use illuminatedfiber tips or LEDs positioned appropriately (side scatter) orepi-illumination (back scatter) along with colored stained forwardscatter methods such as Rheinberg illumination.

Polarization techniques work well with polarization sensitive specimenssuch as minerals and are an essential element of a petrographicmicroscope. Polarization staining does not work well with mostmicroorganisms as most do not significantly affect the lightpolarization. However, combining polarization with interference ordark-field techniques can provide some very interesting and artisticimagery, particularly when protozoa and sand grain crystals are mixed.

Interference staining techniques convert phase (optical time delay)differences into optical intensity and color differences. Transparentmicro fauna and flora become highly visible with the intensity or colordifferences produced by varying optical propagation delays. There aremany variations including phase contrast (Phaco), differentialinterference contrast (DIC), Michelson interferometers and interferencephase. Hardware for interference staining tends to be more expensive andcomplex to use than the other optical staining methods but can imagemicroscopic structures that would otherwise be invisible.

Interference phase color difference and contrast staining were used forthese experiments as this accessory was available. Fortunately, theinterference phase system is well matched to this application for avariety of reasons including good background light suppression as neededfor RAS, native support for a broad spectrum of optical stainingtechniques and direct compatibility with the water contact objectivelenses. Interference phase staining operates on the principle of usingpolarization to control the mutual interference between two light paths,one coming directly through the specimen and the other diffracted andbrought into interference with the first path. Depending upon thepresence or absence of a narrow band optical filter, it provides eithercontrast or color difference staining. Phase differences can becalibrated in terms of the interference colors.

The interference phase hardware consists of two main units, a specialcondenser lens assembly and an interference unit that fits between themicroscope body and trinocular viewing head. The condenser lens assemblywith annular rings is like those used for phase contrast imaging. Thefairly-complex interference unit includes an adjustable polarizer tovary the phase difference between the two paths, a ¼ wave plate, a setof phase rings on an indexed slider and an adjustable analyzer to varythe interference mixing ratio (contrast difference) along with severalother lenses and folding prisms. The accessories include a centeringtelescope used for alignment and a narrow band optical filter used forcontrast imaging. The RAS detector assembly with a beam splitter ismounted between the interference unit and trinocular head. A narrow bandor band reject filter in the RAS optical path only improves the acousticsensitivity by reducing background illumination levels usinginterference phase contrast while simultaneously using the colordifference mode in the visual channel. The interference phase system isquite versatile and adaptable and can be reconfigured to other opticalstaining methods including bright-field, dark-field, obliqueillumination, phase contrast and polarization along with a broad varietyof interesting intermediate combinations.

So far, the best microRAS results have been obtained with opticalstaining techniques that produce a dark background to minimize shotnoise. These include annular and oblique forward scatter dark-fieldillumination and interference phase contrast. Simultaneous interferencephase color difference imaging is possible with the optical bandpassfilter removed from the visual channel. This results in a pleasingvisual image yet with the desirable dark background in the RAS sensorpath. Interference phase also provides a better visibility of certainprotozoa structures than the more conventional dark-field methods.Interference phase requires brighter illumination than dark field due topolarization and other light losses.

There were two fundamental requirements for this project, i.e., first,the fauna and flora need to be alive, healthy and free to move, andsecond, the microscope should be capable of producing artisticallyinteresting cinema quality video imagery along with multichannel sound.The prototype microRAS system was based on a Nikon S-Ke opticalmicroscope with adaptations to include high definition video recordingand multichannel in situ audio recording.

-   -   LED Illumination: The microscope was modified to use LED        illumination. Advantages of LED illumination include the higher        brightness needed to better support the multiple optical paths,        minimal UV, IR and heat as needed for the micro-organism health        and a longer lamp life. The existing microscope lamp was        replaced with a 1.1 watt BA15D/1142 LED spot light bulb with a        6000K color temperature and 85 lumen output. This lamp output is        sufficient for dark-field but a brighter lamp is desirable for        interference phase. The lamp and high definition video camera        were operated from regulated 12 volt power supply. Although this        lamp internally uses a switching power converter, it does not        produce any audio frequency intensity noise. There is some color        variation across the LED emitter but it is acceptable for this        prototype system.    -   Microscope objective lenses: Microscope objective lenses in the        4× to 40× range were used to view the free-swimming fauna and        flora. Standard microscope objective lenses are designed for        viewing through air into a thin glass cover slip that is in        direct contact with the specimen. Several problems occur when        instead using these objectives to view directly into a water        environment. The water depth must be limited to prevent direct        water immersion and the variable water optical path length along        with a lack of a glass cover slip can cause spherical aberration        at higher magnifications. A varying surface curvature at the air        to water interface is produced by surface tension and water        evaporation. This will cause a misalignment of the optical        staining hardware resulting in a loss of visual image quality        and acousto-optical modulation depth. Even at lower powers, the        microscope will need to be frequently refocused and the optical        staining hardware will need frequent realignment. A better        choice is to use microscope objectives specifically designed for        direct water immersion, completely eliminating an air to water        boundary. With these lenses, there is no change in optical        performance with water depth, evaporation or any destabilization        of the optical staining process. The prototype system used a        pair of Nikon 10× and 20× water contact objectives (77798        & 77803) with an 8 mm depth capability.    -   Optical staining: Forward scatter (annular dark-field & oblique)        illumination and interference phase optical staining methods        used in these experiments. Dark field and oblique illumination        was obtained, either using a Nikon Achromatic 5 lens condenser        with iris, oblique illumination slider and filter holder (77894)        augmented a set of dark-field diaphragms or with the annular        ring condenser assembly from the interference phase accessory        (77000). Interference phase imaging required using both the        special condenser lens assembly and the interference unit that        fits between the microscope body and trinocular head. All of        these optical staining methods are compatible with both the        conventional and water contact objective lenses.    -   Trinocular viewing system and camera port: A Nikon type U        trinocular head (77759) provides direct binocular viewing, a        camera port or a 50% combination of the two. Direct binocular        viewing, combined with RAS audio in headphones provides an        immersive experience with a sharp, wide angle HDR view and was        commonly used for general exploration and when tracking fast        moving fauna. One of the binocular eyepieces includes a reticule        for camera and RAS targeting (77880). The eyepiece port is also        used with the centering telescope to align the optical staining        hardware. Some microscopes include a switchable Bertrand lens        which eliminates the need for a separate centering telescope.    -   High definition video imaging: A Marshall Electronics CV342-CS        C-mount 1080P30 video camera was mounted directly to the        trinocular camera port using a custom C-mount adapter. This        camera provided a 430 micron horizontal field of view when using        the conventional 10× air objective lens, 400 microns with the        10× UW lens and 200 microns with the 20× UW lens. The camera        HD-SDI output was connected to a Sound Devices Pix240i broadcast        digital video recorder that also records two RAS audio channels.    -   XY specimen positioning stage: The micro-aquatic environment was        contained within a 60 mm diameter optical grade Petri dish        (Corning 25010). The microscope included a Nikon type R        rectangular mechanical stage (77765) to position the Petri dish.        The Nikon type R microscope stage was modified to include a        small joystick handle making it much easier to follow moving        subjects than the normal method of stage positioning with a        coaxial XY knob pair.    -   Remote acoustic sensor (RAS) assembly: The microRAS assembly        fits between the microscope and trinocular viewing head. This        prototype included a beam splitter, a focusing lens, the        photo-receiver and the associated support electronics. A modular        optical bench was used so that a variety of experimental        configurations could be readily explored. A single radiometer        channel provides monophonic audio while a 4 channel version        provides stereo and quadraphonic outputs. Additional details        about the RAS photo-receiver are described above and in the        co-pending application Ser. No. 15/230,369, the subject matter        of which is incorporated herein by reference. A computer        controlled calibration system supports RAS alignment,        calibration and performance measurements. The RAS outputs are        connected to a Sound Devices 633 audio recorder that supports up        to 6 audio channels with time code synchronization. A        stereophonic audio mix is then passed to the Sound Devices        Pix240i video recorder.

It should be appreciated that the AFMS photodetector by itself is onlysensitive to audio frequency light intensity variations. The lightreceived by the AFMS detector includes both unmodulated light thatcarries no acoustic information and acousto-optically modulated lightthat carries acoustic information. It is desirable to minimizeinterference from unmodulated light so as to maximize the sensitivity tothe acousto-optically modulated light. Optical staining improves theacoustic sensitivity by reducing unmodulated light and increasing theintensity modulation depth of the acoustically modulated light. Thereare three basic optical staining approaches used for acousto-opticsensing:

-   -   Intensity modulation enhancement—Unmodulated light that carries        no acoustic information is reduced by redirecting it away from        the sensor. This includes dark field and oblique illumination        techniques. The sensor now primarily receives acousto-optically        modulated light.    -   Conversion of acoustically modulated optical path length        variations into light intensity variations—The AFMS device can        be made sensitive to acoustically modulated path length        variations within an organism. For example, a vibrating surface        within the organism will change the optical path length or time        delay at an audio rate. Various forms of interferometry can        convert the acoustically modulated path length difference        between the organism acoustic vibrations and a reference optical        path into a light intensity signal that is usable by the AFMS        device. Examples of suitable interferometry techniques include        phase contrast methods, interference phase, Nomarski, etc. These        approaches convert phase shifts in light passing through a        transparent organism to brightness changes in the image. The        time delays and corresponding phase shifts themselves are        invisible, but become detectable by AFMS when converted into        light intensity variations.    -   Conversion of acoustically modulated polarization into light        intensity variations—Adding a pair of crossed polarizers to the        optical system will suppress steady state illumination. A        vibrating object between the polarizers that changes the        polarization state will change the light intensity with the AFMS        device is sensitive to at an audio rate.

The RAS signals from the micro-aquatic environment are predominantly lowfrequency with most of the energy below 500 Hz. Many of the signals areproduced by fauna swimming, cilia and feeding motions. Multichannelhaptics corresponding to the spatially distributed RAS detectors canprovide haptic feedback as to fauna motion and activity. Anotherinteresting idea to consider is that the haptics and audio can bebidirectional, creating a so-called Protozoa petting zoo.

Initial experiments used a Texas Instruments DRV2605EVM-CT hapticsevaluation kit. The evaluation module (EVM) audio input was connected toone of the RAS audio mix outputs. This provided a simple and effectivefirst attempt to feel the vibrations produced by the micro-aquatic life.Rotifers with their periodic cilia retractions and other motionsprovided a good demonstration. This was followed by further experimentswith a more powerful small electromagnetic shaker. A first attempt wasmade at using haptic feedback to the micro-fauna but this resulted in ahaptic/RAS feedback loop.

The microorganisms are unconstrained and free to move in threedimensions. Some fauna including the sessile Cothurnia rotifer arestationary. Others move slowly making them relatively easy to follow.Some of the micro fauna move very fast with erratic paths. Manualtracking of these animals is difficult to impossible, and may requireautomated tracking (discussed above). The fastest animals produce asingle RAS pop sound and then completely vanish from the scene in theblink of an eye (less than one video frame time). Wide-area,high-resolution, high-speed, wide-dynamic range video cameras are onepossibility but technology will need to be further developed. Solvingthe high-speed motion problem requires either a faster tracking systemor protozoa motion constraints. Manual XY stage tracking works to apoint but faster motion would require some type of an auto tracker,either video or acoustically based. As a side note, additional sensoryinformation provided by RAS has been effectively used to acousticallyaid visual tracking of rapidly moving aerospace vehicles (see Ser. No.15/230,369, the subject matter of which is incorporated herein byreference).

Constraining or otherwise slowing micro-fauna motion is another option.This can be done in various ways with varying degrees of micro faunafriendliness. Simple appropriate methods include adding more naturallyoccurring bio-materials or other barrier material or reducing the amountof water in the micro-aquatic environment. The fauna seem to like tohide and slowdown in plant life regions and that also helps to ease thetracking requirements. But then the fauna is harder to find and clearlyphotograph. Many times, sessile rotifers were first located acousticallyand then visually found well hidden in their environments. Sometimeslarger transparent fauna eat the smaller protozoa leaving the preynaturally well constrained. Other active and dynamic constraints may bepossible including piezoelectric devices, micromanipulators, galvanicand electrolytic systems. Dynamic active constraints could be used forhaptic feedback and interaction with the micro-fauna.

In testing the invention, the general philosophy was to observeorganisms in as close to normal environmental conditions as possible. Nochemicals were added and the fauna and flora were not unnaturallyconstrained other than by the water volume within the 60 mm diameterPetri dish. Light and heat levels were controlled to minimizeenvironmental stresses.

This study focused on fresh water micro-aquatic environments. Thisenvironment was chosen because of the convenient availability of a widediversity of microscopic fauna and flora. Also important was that anin-water environment is acoustically isolated with minimal acousticcoupling to an air environment. Other fluid environments include seawater micro-aquatic environments and body fluids such as dynamic whiteblood cell activity. Other micro-environments, such as air, are alsointeresting. Audio frequency vibrations, such as insect wing motion, areweakly coupled as sound waves and are easily observable with opticalsensors.

It should be appreciated that the present invention is not limited tothe block diagram shown in FIG. 10, or the experimental details providedabove. The present invention can be used in any number of ways, toaurally (and/or visually) observe any number of specimens. While theinventor made design choices in carrying out his experimentations, thosedesign choices are not limitations of the present invention, and thoseskilled in the art will appreciated that various modifications can bemade, depending on the type and/or number of specimens that are beingobserved, and the specific goals that are to be achieved.

Having thus described several embodiments of a system and method forusing a microscope to aurally observe a specimen in a fluid (e.g.,water, air, etc.), it should be apparent to those skilled in the artthat certain advantages of the system and method have been achieved. Itshould also be appreciated that various modifications, adaptations, andalternative embodiments thereof may be made within the scope and spiritof the present invention. The invention is solely defined by thefollowing claims.

What is claimed is:
 1. A microscope for haptically observing at leastone organism, comprising: a light source; a container comprising a fluidand said at least one organism; and an audio frequency modulationsensing (AFMS) device adapted to convert an optical signal from said atleast one organism into an electrical signal; and a haptic feedbackdevice adapted to convert said electrical signal into at leastvibrations, thereby providing a user with haptic feedback associatedwith said optical signal from said at least one organism.
 2. Themicroscope of claim 1, wherein said haptic feedback device is furtheradapted to selectively provide a second electrical signal to a secondhaptic feedback device in said fluid, thereby allowing for bidirectionalhaptic feedback between said user and said at least one organism.
 3. Themicroscope of claim 2, wherein said second haptic feedback device is ashaker device.
 4. The microscope of claim 2, wherein said second hapticfeedback device is a piezo electric device.
 5. The microscope of claim2, wherein said second haptic feedback device is an electric currentinducing device.
 6. The microscope of claim 1, wherein said AFMS deviceis further adapted to extract aural data from said electrical signal. 7.The microscope of claim 6, wherein said aural data is based on anintensity of audio frequency modulated electromagnetic energy in saidoptical signal received by said AFMS device.
 8. The microscope of claim6, further comprising a video camera for capturing at least one visualof said at least one organism, said at least one visual being output intime synchronization with said aural data;
 9. The microscope of claim 8,further comprising at least one beam splitter for splitting a firstoptical signal from said at least one organism into at least saidoptical signal and a second optical signal, said optical signal beingprovided to said AFMS device, and said second optical signal beingprovided to said video camera.
 10. The microscope of claim 6, whereinsaid electrical signal is used to generate video data that is output intime synchronization with said aural data.
 11. The microscope of claim1, further comprising an optical staining device for decreasing theamount of light unmodulated by said at least one organism in saidoptical signal received by said AFMS device and increasing asignal-to-noise ratio (“SNR”) of said optical signal received by saidAFMS device.
 12. The microscope of claim 11, wherein said opticalstaining device comprises a dark-field illuminator.
 13. The microscopeof claim 1, further comprising an objective lens for magnifying saidoptical signal prior to said optical signal being provided to said AFMSdevice.
 14. The microscope of claim 13, wherein said objective lens isconfigured to be submerged at least partially in said fluid.
 15. Themicroscope of claim 6, wherein said AFMS device comprises a plurality ofsensors, a first one of said plurality of sensors is used to convertsaid optical signal into said electrical signal, a second one of saidplurality of sensors is used to convert a second optical signal into asecond electrical signal, and other aural data is extracted from saidsecond electrical signal, wherein said aural data and said other auraldata are used to create stereophonic sound.
 16. A method for using amicroscope to haptically observe at least one organism, comprising:providing at least one organism in a fluid; using a light source togenerate a light; using an audio frequency modulation sensing (AFMS)device to convert an optical signal from said at least one organism intoan electrical signal; and using a haptic device to convert saidelectrical signal into at least vibrations, thereby providing a userwith haptic feedback associated with said optical signal from said atleast one organism.
 17. The method of claim 16, further comprising thestep of selectively providing a second electrical signal to a secondhaptic device in said fluid, thereby allowing bidirectional hapticfeedback between said user and said at least one organism.
 18. Themethod of claim 17, wherein said second haptic feedback device is ashaker device.
 19. The method claim 17, wherein said second hapticfeedback device is a piezo electric device.
 20. The method of claim 17,wherein said second haptic feedback device is an electric currentinducing device.
 21. The method of claim 16, wherein said step of usingsaid AFMS device to convert said optical signal from said at least oneorganism into said electrical signal further comprises extracting auraldata from said electrical device, said aural data being based on audiofrequency light intensity fluctuations in said optical signal.
 22. Themethod of claim 21, further comprising the step of using a video camerato capture at least one visual of said at least one organism, said atleast one visual being output in time synchronization with said auraldata.
 23. The method of claim 22, further comprising the step ofsplitting a first optical signal from said at least one organism intosaid optical signal and a second optical signal, said second opticalsignal being provided, at least indirectly, to said video camera. 24.The method of claim 21, further comprising the step of using saidelectrical signal to generate a video data, said video data being outputin time synchronization with said aural data.
 25. The method of claim16, using an optical staining device to decrease the amount of lightunmodulated by said at least one organism in said optical signalreceived by said AFMS device and increase a signal-to-noise ratio(“SNR”) of said optical signal received by said AFMS device.
 26. Asystem for haptically observing at least one organism, comprising: alight source; a living organism; at least one sensor for converting anoptical signal from said at least one organism into an electricalsignal; a haptic feedback device adapted to convert said electricalsignal into at least vibrations, thereby providing a user with hapticfeedback associated with said optical signal from said at least oneorganism.
 27. The system of claim 26, wherein said haptic feedbackdevice if further adapted to provide a second electrical signal to asecond haptic feedback device within said fluid, thereby allowing forbidirectional haptic feedback between said user and said at least oneorganism.
 28. The system of claim 26, further comprising an opticalstaining device for decreasing an amount of light unmodulated by saidliving organism from entering said AFMS device.